Image processing apparatus using artificial intelligence super-resolution circuit and follow-up resize circuit for image resizing

ABSTRACT

An image processing apparatus includes a super-resolution (SR) circuit and a resizer circuit. The SR circuit performs an SR operation upon a first image to generate a second image, wherein a resolution of the second image is not lower than a resolution of the first image, and the SR operation is based, at least in part, on one or more artificial intelligence (AI) models. The resizer circuit performs a resize operation upon the second image to generate a third image, wherein a resolution of the third image is not lower than the resolution of the second image, and no AI model is involved in the resize operation.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional application No. 62/987,962, filed on Mar. 11, 2020 and incorporated herein by reference.

BACKGROUND

The present invention relates to an image processing technique, and more particularly, to an image processing apparatus using an artificial intelligence super-resolution circuit and a follow-up resizer circuit for image resizing.

Modern image display devices typically enhance the images before they are displayed. Some devices can perform super-resolution for image resizing. For example, a super-resolution operation can be used to up-scale a low resolution image (e.g., an image with 720×480 pixels) to a high resolution image (e.g., an image with 3840×2160 pixels). The super-resolution function is a key feature on a television and a smartphone. However, a typical edge device, such as the television or the smartphone, has limited computing power due to strict requirements on power consumption, thermal performance and hardware cost. A super-resolution function with image enhancement may have limited scaling capability due to limited computing power of the typical edge device. Thus, there is a need for an innovative design that has a super-resolution function integrated in an image processing pipeline and is capable of providing arbitrary-ratio image resizing.

SUMMARY

One of the objectives of the claimed invention is to provide an image processing apparatus using an artificial intelligence super-resolution circuit and a follow-up resizer circuit for image resizing.

According to a first aspect of the present invention, an exemplary image processing apparatus is disclosed. The exemplary image processing apparatus includes a super-resolution (SR) circuit and a resizer circuit. The SR circuit is arranged to perform an SR operation upon a first image to generate a second image, wherein a resolution of the second image is not lower than a resolution of the first image, and the SR operation is based, at least in part, on one or more artificial intelligence (AI) models. The resizer circuit is arranged to perform a resize operation upon the second image to generate a third image, wherein a resolution of the third image is not lower than the resolution of the second image, and no AI model is involved in the resize operation.

According to a second aspect of the present invention, an exemplary image processing apparatus is disclosed. The exemplary image processing apparatus includes a super-resolution (SR) circuit and a resizer circuit. The SR circuit is arranged to perform an SR operation upon a first image to generate a second image, wherein a resolution of the second image is not lower than a resolution of the first image, the SR circuit supports fixed scaling ratios only, and any scaling ratio employed by the SR operation is selected from the fixed scaling ratios. The resizer circuit is arranged to perform a resize operation upon the second image to generate a third image, wherein a resolution of the third image is not lower than the resolution of the second image, and any scaling ratio employed by the resize operation is an arbitrary scaling ratio supported by the resizer circuit.

These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating some examples of arbitrary-ratio scaling with artificial intelligence aided image enhancement according to an embodiment of the present invention.

FIG. 3 is a diagram illustrating a raster scan order employed by a super-resolution circuit and/or an image pre-processing circuit according to an embodiment of the present invention.

DETAILED DESCRIPTION

Certain terms are used throughout the following description and claims, which refer to particular components. As one skilled in the art will appreciate, electronic equipment manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not in function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is coupled to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.

FIG. 1 is a diagram illustrating an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 100 includes an image pre-processing circuit 106, a super-resolution (SR) circuit 102, a resizer circuit 104, and a result collection buffer 108. In this example, the image processing apparatus 100 may be a part of a system on a chip (SoC), and the SoC may be used by an edge device such as a television, a smartphone, or any playback device capable of processing and displaying images and/or videos. The image pre-processing circuit 106, the SR circuit 102, the resizer circuit 104, and the result collection buffer 108 may integrated in a same chip 10. The image pre-processing circuit 106 is arranged to receive an input image IMG_IN from an input port P1 of the image processing apparatus 100, and perform an image pre-processing operation upon the input image IMG_IN to generate a pre-processed image IMG_PP. For example, the image pre-processing circuit 106 may include a noise reduction (NR) filter, such that NR is applied to the input image IMG_IN by the image pre-processing operation. However, this is for illustrative purposes only, and is not meant to be a limitation of the present invention. In practice, the image pre-processing circuit 106 may be configured to apply any suitable image enhancement to the input image IMG_IN before the SR operation. A resolution of the pre-processed image IMG_PP may be equal to or different from a resolution of the input image IMG_IN, depending upon the actual image enhancement technique employed by the image pre-processing circuit 106.

The SR circuit 102 is arranged to perform an SR operation upon the pre-processed image IMG_PP to generate an SR image IMG_SR, wherein a resolution of the SR image IMG_SR is not lower than a resolution of the pre-processed image IMG_PP. In a case where a first scaling ratio S1 (S1=1×) is employed by the SR operation, the resolution of the SR image IMG_SR is equal to the resolution of the pre-processed image IMG_PP. For example, the SR image IMG_SR and the pre-processed image IMG_PP may have the same number of pixels for the same display size. In another case where a second scaling ratio S2 (S2>1×) is employed by the SR operation, the resolution of the SR image IMG_SR is higher than the resolution of the pre-processed image IMG_PP. For example, the SR image IMG_SR may have A pixels and the pre-processed image IMG_PP may have B pixels for the same display size, where A is larger than B.

In this embodiment, the SR circuit 102 performs the SR operation with aid of artificial intelligence (AI). Specifically, the SR operation is based, at least in part, on one or more AI models. Hence, the SR circuit 102 can perform the SR operation with image enhancement. Examples of AI models include, but are not limited to, convolutional neural network (CNN) models, machine learning models, deep learning models, etc. Furthermore, the SR circuit 102 may be implemented by an AI processing unit (APU), a graphic processing unit (GPU), an application-specific processor, or a general-purpose processor.

The resizer circuit 104 is arranged to perform a resize operation upon the SR image IMG_SR to generate an output image IMG_OUT, wherein a resolution of the output image IMG_OUT is not lower than the resolution of the SR image IMG_SR. For example, the resize operation with a scaling ratio larger than 1 is performed to up-scale a lower resolution image to a higher resolution image, such that the resolution of the output image IMG_OUT is higher than the resolution of the SR image IMG_SR, that is, the output image IMG_OUT may have C pixels and the SR image IMG_SR may have D pixels for the same display size, where C is larger than D. For another example, the resolution of the SR image IMG_SR is equal to a target resolution, and the resize operation with a scaling ratio equal to 1 is applied to the SR image IMG_SR, such that the resolution of the output image IMG_OUT is equal to the resolution of the SR image IMG_SR, that is, the output image IMG_OUT and the SR image IMG_SR may have the same number of pixels for the same display size. In this embodiment, the resizer circuit 104 performs the resize operation without aid of AI, such that no AI model is involved in the resize operation.

The result collection buffer 108 is arranged to buffer pixels of the output image IMG_OUT, and output buffered pixels of the output image IMG_OUT via an output port P6 of the image processing apparatus 100.

Since the SR circuit 102 is designed to perform artificial intelligence super-resolution (AISR), no arbitrary-ratio scaling can be achieved by the SR circuit 102. For example, one set of trained CNN parameters is required by the SR operation for applying one scaling ratio, and the AI engine (e.g., SR circuit 102) can acquire only a limited number of trained CNN parameter sets. Hence, the SR circuit 102 supports fixed scaling ratios only, and any scaling ratio employed by the SR operation is selected from the fixed scaling ratios.

In contrast to the SR circuit 102, the resizer circuit 104 is designed to perform the resize operation without aid of AI. Hence, arbitrary-ratio scaling can be achieved by the resizer circuit 104. That is, any scaling ratio employed by the resize operation is an arbitrary scaling ratio supported by the resizer circuit 104.

As mentioned above, the SR circuit 102 performs the SR operation with AI-aided image enhancement, but lacks scaling flexibility due to fixed-ratio scaling; and the resize circuit 104 performs the arbitrary-ratio resize operation without AI-aided image enhancement. With the help of the resize circuit 104 cascaded to the SR circuit 102, arbitrary-ratio scaling with AI-aided image enhancement can be achieved. FIG. 2 is a diagram illustrating some examples of arbitrary-ratio scaling with AI-aided image enhancement according to an embodiment of the present invention. Suppose that the SR circuit 102 supports four fixed scaling ratios 1×, 2×, 3×, and 4× only, the pre-processed image IMG_PP has W pixels in a width direction and H pixels in a height direction, and a target resolution of the output image IMG_OUT is an ultra high definition (UHD) resolution. Ina first case where the resolution W×H of the pre-processed image IMG_PP is higher than a full high definition (FHD) resolution, the SR circuit 102 performs the fixed-ratio SR operation with the fixed scaling ratio 1× upon the pre-processed image IMG_PP, and then the resizer circuit 104 refers to discrepancy between a resolution of the pre-processed image IMG_PP and the target UHD resolution to program a scaling ratio adaptively, and performs the arbitrary-ratio resize operation with the adaptively programed scaling ratio upon the SR image IMG_SR to generate the output image IMG_OUT with the target UHD resolution.

In a second case where the resolution W×H of the pre-processed image IMG_PP is higher than a high definition (HD) resolution but not higher than the FHD resolution, the SR circuit 102 performs the fixed-ratio SR operation with the fixed scaling ratio 2× upon the pre-processed image IMG_PP, and then the resizer circuit 104 refers to discrepancy between a resolution of the pre-processed image IMG_PP and the target UHD resolution to program a scaling ratio adaptively, and performs the arbitrary-ratio resize operation with the adaptively programmed scaling ratio upon the SR image IMG_SR to generate the output image IMG_OUT with the target UHD resolution.

In a third case where the resolution W×H of the pre-processed image IMG_PP is higher than 960×540 but not higher than the HD resolution, the SR circuit 102 performs the fixed-ratio SR operation with the fixed scaling ratio 3× upon the pre-processed image IMG_PP, and then the resizer circuit 104 refers to discrepancy between a resolution of the pre-processed image IMG_PP and the target UHD resolution to program a scaling ratio adaptively, and performs the arbitrary-ratio resize operation with the adaptively programmed scaling ratio upon the SR image IMG_SR to generate the output image IMG_OUT with the target UHD resolution.

In a fourth case where the resolution W×H of the pre-processed image IMG_PP is not higher than 960×540, the SR circuit 102 performs the fixed-ratio SR operation with the fixed scaling ratio 4× upon the pre-processed image IMG_PP, and then the resizer circuit 104 refers to discrepancy between a resolution of the pre-processed image IMG_PP and the target UHD resolution to program a scaling ratio adaptively, and performs the arbitrary-ratio resize operation with the adaptively programmed scaling ratio upon the SR image IMG_SR to generate the output image IMG_OUT with the target UHD resolution.

As illustrated in FIG. 2, arbitrary-ratio scaling with AI-aided image enhancement can be achieved by applying arbitrary-ratio resizing to an output of fixed-ratio AISR. It should be noted that the up-scaling operations shown in FIG. 2 are for illustrative purposes only, and are not meant to be limitations of the present invention. In practice, a combination of the SR circuit 102 and the follow-up resizer circuit 104 is capable of performing scaling from any source resolution to any target resolution.

When the image processing apparatus 100 is employed by an edge device such as a television or a smartphone, the SR circuit 102 is an AI engine with limited computing power due to strict requirements on power consumption, thermal performance and hardware cost. In this embodiment, the SR circuit 102 is cascaded to the image pre-processing circuit 106, and the image pre-processing circuit 106 applies image enhancement (e.g., NR) to the input image IMG_IN before AISR. Since the SR circuit 102 applies AISR to the pre-processed image IMG_PP with enhanced image quality, the image quality of the SR image IMG_SR can be improved.

It should be noted that the image pre-processing circuit 106 may be optional. In an alternative design, the image processing apparatus 100 may be modified to omit the image pre-processing circuit 106, and the SR circuit 102 is modified to perform the SR operation upon the input image IMG_IN to generate the SR image IMG_SR. In practice, any image processing apparatus using a fixed-ratio SR circuit (e.g., an SR circuit with aid of AI) and an arbitrary-ratio resizer circuit (e.g., a resizer circuit without aid of AI) to achieve arbitrary-ratio scaling with image enhancement (e.g., arbitrary-ratio scaling with AI-aided image enhancement) falls within the scope of the present invention.

In some embodiments of the present invention, one or both of the image pre-processing circuit 106 and the SR circuit 102 may employ raster scan based processing for making the image processing apparatus 100 have low latency and low bandwidth. For example, in the embodiment shown in FIG. 1, each of the image pre-processing circuit 106 and the SR circuit 102 employs raster scan based processing

FIG. 3 is a diagram illustrating a raster scan order employed by the SR circuit 102 and/or the image pre-processing circuit 106 according to an embodiment of the present invention. Suppose that a resolution of an image IMG is M×N, where M and N are positive integers. Hence, the image IMG includes a plurality of pixel lines L₁, L₂, . . . , L_(N-1), L_(N), and each of the pixel lines L₁-L_(N) includes a plurality of pixels P₁, P₂, P₃, P₄, . . . , P_(M-1), P_(M).

Consider a case where the image IMG is an input image to be processed by one image processing pipeline stage. In accordance with a raster scan order R, pixels of the same pixel line are sequentially processed from the left-most pixel P₁ to the right-most pixel P_(M), and pixel lines of the same image are sequentially processed from the top-most pixel line L₁ to the bottom-most pixel line L_(N).

Consider another case where the image IMG is an output image generated from one image processing pipeline stage. In accordance with the raster scan order R, pixels of the same pixel line are sequentially output from the left-most pixel P₁ to the right-most pixel P_(M), and pixel lines of the same image are sequentially output from the top-most pixel line L₁ to the bottom-most pixel line L_(N).

When the image IMG shown in FIG. 3 is the SR image IMG_SR, the SR circuit 102 is further arranged to output pixels of the SR image IMG_SR in a raster scan order, where pixels at each pixel line are sequentially output in a left-to-right order, and pixel lines are sequentially output in a top-to-bottom order.

When the image IMG shown in FIG. 3 is the pre-processed image IMG_PP, the image pre-processing circuit 106 is further arranged to output pixels of the pre-processed image IMG_PP in a raster scan order, where pixels at each pixel line are sequentially output in a left-to-right order, and pixel lines are sequentially output in a top-to-bottom order.

Due to the inherent characteristic of the raster scan based processing, an output port P4 of the SR circuit 102 can be directly linked to an input port P5 of the resizer circuit 104, that is, the image processing apparatus 100 has no frame buffer (which is used to store one full image) or block buffer (which is used to store an m×n block in an image, where the m×n block has m pixels in a width direction of the image and n pixels in a height direction of the image, and n is larger than 1) coupled between the output port P4 of the SR circuit 104 and the input port P5 of the resizer circuit 104. Similarly, an output port P2 of the image pre-processing circuit 106 can be directly linked to an input port P3 of the SR circuit 102, that is, the image processing apparatus 100 has no frame buffer (which is used to store one full image) or block buffer (which is used to store an m×n block in an image, where the m×n block has m pixels in a width direction of the image and n pixels in a height direction of the image, and n is larger than 1) coupled between the output port P2 of the image pre-processing circuit 106 and the input port P3 of the SR circuit 102.

A frame buffer/block buffer is generally implemented by a dynamic random access memory (DRAM) or a static random access memory (SRAM). As a result, the frame buffer/block buffer occupies a part of the chip area and introduces image data transfer latency. Since the SR circuit 102 and/or the image pre-processing circuit 106 employ raster scan order processing and the image processing apparatus 100 does not need to use a frame buffer/block buffer between image pre-processing circuit 106 and SR circuit 102 and/or a frame buffer/block buffer between SR circuit 102 and resizer circuit 104, a low cost, low latency and low bandwidth processing apparatus can be achieved.

It should be noted that any image processing apparatus using a fixed-ratio SR circuit (e.g., an SR circuit with aid of AI) and an arbitrary-ratio resizer circuit (e.g., a resizer circuit without aid of AI) to achieve arbitrary-ratio scaling with image enhancement (e.g., arbitrary-ratio scaling with AI-aided image enhancement) falls within the scope of the present invention. For example, the image processing apparatus 100 shown in FIG. 1 may be modified to have a frame buffer/block buffer coupled between image pre-processing circuit 106 and SR circuit 102 and/or a frame buffer/block buffer coupled between SR circuit 102 and resizer circuit 104. The same objective of providing arbitrary-ratio scaling with image enhancement (e.g., arbitrary-ratio scaling with AI-aided image enhancement) by a fixed-ratio SR circuit (e.g., an SR circuit with aid of AI) and an arbitrary-ratio resizer circuit (e.g., a resizer circuit without aid of AI) is achieved.

Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims. 

What is claimed is:
 1. An image processing apparatus comprising: a super-resolution (SR) circuit, arranged to perform an SR operation upon a first image to generate a second image, wherein a resolution of the second image is not lower than a resolution of the first image, and the SR operation is based, at least in part, on one or more artificial intelligence (AI) models; and a resizer circuit, arranged to perform a resize operation upon the second image to generate a third image, wherein a resolution of the third image is not lower than the resolution of the second image, and no AI model is involved in the resize operation.
 2. The image processing apparatus of claim 1, wherein the second image comprises a plurality of pixel lines each having a plurality of pixels, and the SR circuit is further arranged to output pixels of the second image in a raster scan order, where the plurality of pixels at each pixel line are sequentially output in a left-to-right order, and the plurality of pixel lines are sequentially output in a top-to-bottom order.
 3. The image processing apparatus of claim 1, wherein an output port of the SR circuit is directly linked to an input port of the resizer circuit.
 4. The image processing apparatus of claim 1, wherein the image processing apparatus comprises no frame buffer or block buffer coupled between an output port of the SR circuit and an input port of the resizer circuit.
 5. The image processing apparatus of claim 1, further comprising: an image pre-processing circuit, arranged to perform an image pre-processing operation upon a fourth image to generate the first image.
 6. The image processing apparatus of claim 5, wherein the first image comprises a plurality of pixel lines each having a plurality of pixels, and the image pre-processing circuit is further arranged to output pixels of the first image in a raster scan order, where the plurality of pixels at each pixel line are sequentially output in a left-to-right order, and the plurality of pixel lines are sequentially output in a top-to-bottom order.
 7. The image processing apparatus of claim 5, wherein an output port of the image pre-processing circuit is directly linked to an input port of the SR circuit.
 8. The image processing apparatus of claim 5, wherein the image processing apparatus comprises no frame buffer or block buffer coupled between an output port of the image pre-processing circuit and an input port of the SR circuit.
 9. The image processing apparatus of claim 5, wherein the image pre-processing operation comprises noise reduction.
 10. The image processing apparatus of claim 1, wherein the SR circuit and the resizer circuit are integrated in a same chip.
 11. An image processing apparatus comprising: a super-resolution (SR) circuit, arranged to perform an SR operation upon a first image to generate a second image, wherein a resolution of the second image is not lower than a resolution of the first image, the SR circuit supports fixed scaling ratios only, and any scaling ratio employed by the SR operation is selected from the fixed scaling ratios; and a resizer circuit, arranged to perform a resize operation upon the second image to generate a third image, wherein a resolution of the third image is not lower than the resolution of the second image, and any scaling ratio employed by the resize operation is an arbitrary scaling ratio supported by the resizer circuit.
 12. The image processing apparatus of claim 11, wherein the second image comprises a plurality of pixel lines each having a plurality of pixels, and the SR circuit is further arranged to output pixels of the second image in a raster scan order, where the plurality of pixels at each pixel line are sequentially output in a left-to-right order, and the plurality of pixel lines are sequentially output in a top-to-bottom order.
 13. The image processing apparatus of claim 11, wherein an output port of the SR circuit is directly linked to an input port of the resizer circuit.
 14. The image processing apparatus of claim 11, wherein the image processing apparatus comprises no frame buffer or block buffer coupled between an output port of the SR circuit and an input port of the resizer circuit.
 15. The image processing apparatus of claim 11, further comprising: an image pre-processing circuit, arranged to perform an image pre-processing operation upon a fourth image to generate the first image.
 16. The image processing apparatus of claim 15, wherein the first image comprises a plurality of pixel lines each having a plurality of pixels, and the image pre-processing circuit is further arranged to output pixels of the first image in a raster scan order, where the plurality of pixels at each pixel line are sequentially output in a left-to-right order, and the plurality of pixel lines are sequentially output in a top-to-bottom order.
 17. The image processing apparatus of claim 15, wherein an output port of the image pre-processing circuit is directly linked to an input port of the SR circuit.
 18. The image processing apparatus of claim 15, wherein the image processing apparatus comprises no frame buffer or block buffer coupled between an output port of the image pre-processing circuit and an input port of the SR circuit.
 19. The image processing apparatus of claim 15, wherein the image pre-processing operation comprises noise reduction.
 20. The image processing apparatus of claim 11, wherein the SR circuit and the resizer circuit are integrated in a same chip. 